We present a retrospective on the state of Embodied AI research. Our analysis focuses on 13 challenges presented at the Embodied AI Workshop at CVPR. These challenges are grouped into three themes: (1) visual navigation, (2) rearrangement, and (3) embodied vision-and-language. We discuss the dominant datasets within each theme, evaluation metrics for the challenges, and the performance of state-of-the-art models. We highlight commonalities between top approaches to the challenges and identify potential future directions for Embodied AI research.

我们介绍了Soundspaces 2.0，这是一个用于3D环境的基于几何的音频渲染的平台。考虑到现实世界环境的3D网格，Soundspaces可以为从任意麦克风位置捕获的任意声音生成高度逼真的声音。它与现有的3D视觉资产一起支持一系列视听研究任务，例如视听导航，映射，源定位和分离以及声学匹配。与现有资源相比，Soundspaces 2.0具有允许连续的空间采样，对新型环境的概括以及可配置的麦克风和材料属性的优点。据我们所知，这是第一个基于几何的声学模拟，它提供了高忠诚和现实主义，同时也足够快地用于体现学习。我们展示了模拟器的属性，并根据现实世界的音频测量进行了基准性能。此外，通过涵盖具体导航和远场自动语音识别的两个下游任务，突出了后者的SIM2REAL性能。 Soundspaces 2.0可公开使用，以促进对感知系统的更广泛研究，这些系统既可以看到和听到。

房间冲动响应（RIR）函数捕获周围的物理环境如何改变听众听到的声音，对AR，VR和机器人技术中的各种应用产生影响。估计RIR的传统方法在整个环境中采用密集的几何形状和/或声音测量值，但我们探讨了如何根据空间中观察到的一组稀疏图像和回声来推断RIR。为了实现这一目标，我们介绍了一种基于变压器的方法，该方法使用自我注意力来构建丰富的声学环境，然后通过跨注意来预测任意查询源接收器位置的河流。此外，我们设计了一个新颖的训练目标，该目标改善了RIR预测与目标之间的声学​​特征中的匹配。在使用3D环境的最先进的视听模拟器的实验中，我们证明了我们的方法成功地生成了任意RIR，优于最先进的方法，并且在与传统方法的主要背离中 - 以几种方式概括新的环境。项目：http：//vision.cs.utexas.edu/projects/fs_rir。

我们介绍了视觉匹配任务，其中音频剪辑被转换为听起来像是在目标环境中记录的。鉴于目标环境的图像和源音频的波形，目标是重新合成音频，以匹配目标室声音的可见几何形状和材料所建议的。为了解决这一新颖的任务，我们提出了一个跨模式变压器模型，该模型使用视听注意力将视觉属性注入音频并生成真实的音频输出。此外，我们设计了一个自我监督的训练目标，尽管他们缺乏声学上不匹配的音频，但可以从野外网络视频中学习声学匹配。我们证明，我们的方法成功地将人类的言语转化为图像中描绘的各种现实环境，表现优于传统的声学匹配和更严格的监督基线。

弱监督对象本地化（WSOL）旨在仅通过使用图像级标签来学习对象本地化器。基于卷积神经网络（CNN）的技术通常导致突出显示物体的最辨别部分，同时忽略整个对象范围。最近，变压器架构已经部署到WSOL，以捕获具有自我关注机制和多层的Perceptron结构的远程特征依赖性。然而，变压器缺乏CNN所固有的局部感应偏差，因此可以恶化WSOL中的局部特征细节。在本文中，我们提出了一种基于变压器的新型框架，称为LCTR（局部连续性变压器），该框架被称为LCTR（局部连续性变压器），该框架在长期特征依赖项中提高全局特征的本地感知能力。为此，我们提出了一个关系的修补程序注意模块（RPAM），其考虑全球跨补丁信息。我们进一步设计了一个CUE挖掘模块（CDM），它利用本地特征来指导模型的学习趋势，以突出弱局部响应。最后，在两个广泛使用的数据集，即Cub-200-2011和ILSVRC上进行综合实验，以验证我们方法的有效性。

Recently, density map regression-based methods have dominated in crowd counting owing to their excellent fitting ability on density distribution. However, further improvement tends to saturate mainly because of the confusing background noise and the large density variation. In this paper, we propose a Hierarchically Decoupled Network (HDNet) to solve the above two problems within a unified framework. Specifically, a background classification sub-task is decomposed from the density map prediction task, which is then assigned to a Density Decoupling Module (DDM) to exploit its highly discriminative ability. For the remaining foreground prediction sub-task, it is further hierarchically decomposed to several density-specific sub-tasks by the DDM, which are then solved by the regression-based experts in a Foreground Density Estimation Module (FDEM). Although the proposed strategy effectively reduces the hypothesis space so as to relieve the optimization for those task-specific experts, the high correlation of these sub-tasks are ignored. Therefore, we introduce three types of interaction strategies to unify the whole framework, which are Feature Interaction, Gradient Interaction, and Scale Interaction. Integrated with the above spirits, HDNet achieves state-of-the-art performance on several popular counting benchmarks.

Deep learning models can achieve high accuracy when trained on large amounts of labeled data. However, real-world scenarios often involve several challenges: Training data may become available in installments, may originate from multiple different domains, and may not contain labels for training. Certain settings, for instance medical applications, often involve further restrictions that prohibit retention of previously seen data due to privacy regulations. In this work, to address such challenges, we study unsupervised segmentation in continual learning scenarios that involve domain shift. To that end, we introduce GarDA (Generative Appearance Replay for continual Domain Adaptation), a generative-replay based approach that can adapt a segmentation model sequentially to new domains with unlabeled data. In contrast to single-step unsupervised domain adaptation (UDA), continual adaptation to a sequence of domains enables leveraging and consolidation of information from multiple domains. Unlike previous approaches in incremental UDA, our method does not require access to previously seen data, making it applicable in many practical scenarios. We evaluate GarDA on two datasets with different organs and modalities, where it substantially outperforms existing techniques.

The development of social media user stance detection and bot detection methods rely heavily on large-scale and high-quality benchmarks. However, in addition to low annotation quality, existing benchmarks generally have incomplete user relationships, suppressing graph-based account detection research. To address these issues, we propose a Multi-Relational Graph-Based Twitter Account Detection Benchmark (MGTAB), the first standardized graph-based benchmark for account detection. To our knowledge, MGTAB was built based on the largest original data in the field, with over 1.55 million users and 130 million tweets. MGTAB contains 10,199 expert-annotated users and 7 types of relationships, ensuring high-quality annotation and diversified relations. In MGTAB, we extracted the 20 user property features with the greatest information gain and user tweet features as the user features. In addition, we performed a thorough evaluation of MGTAB and other public datasets. Our experiments found that graph-based approaches are generally more effective than feature-based approaches and perform better when introducing multiple relations. By analyzing experiment results, we identify effective approaches for account detection and provide potential future research directions in this field. Our benchmark and standardized evaluation procedures are freely available at: https://github.com/GraphDetec/MGTAB.

As one of the prevalent methods to achieve automation systems, Imitation Learning (IL) presents a promising performance in a wide range of domains. However, despite the considerable improvement in policy performance, the corresponding research on the explainability of IL models is still limited. Inspired by the recent approaches in explainable artificial intelligence methods, we proposed a model-agnostic explaining framework for IL models called R2RISE. R2RISE aims to explain the overall policy performance with respect to the frames in demonstrations. It iteratively retrains the black-box IL model from the randomized masked demonstrations and uses the conventional evaluation outcome environment returns as the coefficient to build an importance map. We also conducted experiments to investigate three major questions concerning frames' importance equality, the effectiveness of the importance map, and connections between importance maps from different IL models. The result shows that R2RISE successfully distinguishes important frames from the demonstrations.

Compressed videos often exhibit visually annoying artifacts, known as Perceivable Encoding Artifacts (PEAs), which dramatically degrade video visual quality. Subjective and objective measures capable of identifying and quantifying various types of PEAs are critical in improving visual quality. In this paper, we investigate the influence of four spatial PEAs (i.e. blurring, blocking, bleeding, and ringing) and two temporal PEAs (i.e. flickering and floating) on video quality. For spatial artifacts, we propose a visual saliency model with a low computational cost and higher consistency with human visual perception. In terms of temporal artifacts, self-attention based TimeSFormer is improved to detect temporal artifacts. Based on the six types of PEAs, a quality metric called Saliency-Aware Spatio-Temporal Artifacts Measurement (SSTAM) is proposed. Experimental results demonstrate that the proposed method outperforms state-of-the-art metrics. We believe that SSTAM will be beneficial for optimizing video coding techniques.